This metric computes the variance of the mean of the latent encoding of the data points in each dimension of the latent space,Var(µφ(x)i), wherexis sampledfromthedataset. For hierarchical architectures the reported number is for thez2 which is the highest stochasticlayer. Toregularize the Exemplar VAE, we used leave-one-out and exemplar sub-sampling. That is why did not compare directly against a mixture model prior in the primary experimental section. Three different architectures are used in the experiments, described below.

We introduce Exemplar VAEs, a family of generative models that bridge the gap between parametric and non-parametric, exemplar based generative models. Exemplar VAE is a variant of VAE with a non-parametric latent prior based on a Parzen window estimator. To sample from it, one first draws a random exemplar from a training set, then stochastically transforms that exemplar into a latent code and a new observation. We propose retrieval augmented training (RAT) as a way to speed up Exemplar VAE training by using approximate nearest neighbor search in the latent space to define a lower bound on log marginal likelihood. To enhance generalization, model parameters are learned using exemplar leave-one-out and subsampling. Experiments demonstrate the effectiveness of Exemplar VAEs on density estimation and representation learning. Importantly, generative data augmentation using Exemplar VAEs on permutation invariant MNIST and Fashion MNIST reduces classification error from 1.17% to 0.69% and from 8.56% to 8.16%.

We introduce Exemplar VAEs, a family of generative models that bridge the gap between parametric and non-parametric, exemplar based generative models. Exemplar VAE is a variant of VAE with a non-parametric latent prior based on a Parzen window estimator. To sample from it, one first draws a random exemplar from a training set, then stochastically transforms that exemplar into a latent code and a new observation. We propose retrieval augmented training (RAT) as a way to speed up Exemplar VAE training by using approximate nearest neighbor search in the latent space to define a lower bound on log marginal likelihood. To enhance generalization, model parameters are learned using exemplar leave-one-out and subsampling.

Additional Feedback: I read all the reviews and the rebuttal. I agree with the authors that the proposed method is different from learned pseudo-exemplars in the embedding space as in VampVAE, and this work uses real exemplars in the image space. However, I am not convinced that randomly sampling exemplars in the data space with some heuristics based on LOO and trivial exemplar subsampling as regularizations on toy datasets is a significant contribution extending the exemplar-based prior in VampVAE. A possible limitation of the proposed Exemplar VAE is that, the generative model might not learn much beyond reconstruction, instead, it only produces some random samples that stay close to epsilon-ball of training data points. It's possible that Exemplar VAE even performs no better than a deterministic autoencoder with tiny Gaussian noise added to latent codes and k-means regularization in the latent space. VampVAE doesn't have this issue.

And since the ideas are rather incremental Meta-reviewer recommendations: The paper is borderline. R2 is considering increasing the score. R4 recommends rejection based on the lack of novelty compared to the VampPrior and that the paper conducts small-scale non-challenging experiments that doesn't require approximate nearest-neighbor search. He proposes to run the method on ImageNet but I believe this cannot be a condition for acceptance since not everyone has the potential for running such experiments. Furthermore, the paper already covers quite a lot in experiments with MNSIT, Omniglot, FashionMNIST and CelebA, and classification. I believe that R4's concern for novelty are successfully addressed in the rebuttal.

We introduce Exemplar VAEs, a family of generative models that bridge the gap between parametric and non-parametric, exemplar based generative models. Exemplar VAE is a variant of VAE with a non-parametric latent prior based on a Parzen window estimator. To sample from it, one first draws a random exemplar from a training set, then stochastically transforms that exemplar into a latent code and a new observation. We propose retrieval augmented training (RAT) as a way to speed up Exemplar VAE training by using approximate nearest neighbor search in the latent space to define a lower bound on log marginal likelihood. To enhance generalization, model parameters are learned using exemplar leave-one-out and subsampling.

This paper presents a framework for exemplar based generative modeling, featuring Exemplar VAEs. To generate a sample from the Exemplar VAE, one first draws a random exemplar from a training dataset, and then stochastically transforms that exemplar into a latent code, which is then used to generate a new observation. We show that the Exemplar VAE can be interpreted as a VAE with a mixture of Gaussians prior in the latent space, with Gaussian means defined by the latent encoding of the exemplars. To enable optimization and avoid overfitting, Exemplar VAE's parameters are learned using leave-one-out and exemplar subsampling, where, for the generation of each data point, we build a prior based on a random subset of the remaining data points. To accelerate learning, which requires finding the exemplars that exert the greatest influence on the generation of each data point, we use approximate nearest neighbor search in the latent space, yielding a lower bound on the log marginal likelihood. Experiments demonstrate the effectiveness of Exemplar VAEs in density estimation, representation learning, and generative data augmentation for supervised learning.